Accelerating Trajectory Generation for Quadrotors Using Transformers

In this work, we address the problem of computation time for trajectory generation in quadrotors. Most trajectory generation methods for waypoint navigation of quadrotors, for example minimum snap/jerk and minimum-time, are structured as bi-level optimizations. The first level involves allocating time across all input waypoints and the second step is to minimize the snap/jerk of the trajectory under that time allocation. Such an optimization can be computationally expensive to solve. In our approach we treat trajectory generation as a supervised learning problem between a sequential set of inputs and outputs. We adapt a transformer model to learn the optimal time allocations for a given set of input waypoints, thus making it into a single step optimization. We demonstrate the performance of the transformer model by training it to predict the time allocations for a minimum snap trajectory generator. The trained transformer model is able to predict accurate time allocations with fewer data samples and smaller model size, compared to a feedforward network (FFN), demonstrating that it is able to model the sequential nature of the waypoint navigation problem.Comment: Accepted at L4DC 202

Augmented Reality Remote Operation of Dual Arm Manipulators in Hot Boxes

In nuclear isotope and chemistry laboratories, hot cells and gloveboxes provide scientists with a controlled and safe environment to perform experiments. Working on experiments in these isolated containment cells requires scientists to be physically present. For hot cell work today, scientists manipulate equipment and radioactive material inside through a bilateral mechanical control mechanism. Motions produced outside the cell with the master control levers are mechanically transferred to the internal grippers inside the shielded containment cell. There is a growing need to have the capability to conduct experiments within these cells remotely. A simple method to enable remote manipulations within hot cell and glovebox cells is to mount two robotic arms inside a box to mimic the motions of human hands. An AR application was built in this work to allow a user wearing a Microsoft HoloLens 2 headset to teleoperate dual arm manipulators by grasping robotic end-effector digital replicas in AR from a remote location. In addition to the real-time replica of the physical robotic arms in AR, the application enables users to view a live video stream attached to the robotic arms and parse a 3D point cloud of 3D objects in their remote AR environment for better situational awareness. This work also provides users with virtual fixture to assist in manipulation and other teleoperation tasks.Comment: Abstract Submitted to the First International Workshop "Horizons of an Extended Robotics Reality" at the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://sites.google.com/view/xr-robotics-iros2022/home?authuser=

Using Augmented Reality to Assess and Modify Mobile Manipulator Surface Repair Plans

Industrial robotics are redefining inspection and maintenance routines across multiple sectors, enhancing safety, efficiency, and environmental sustainability. In outdoor industrial facilities, it is crucial to inspect and repair complex surfaces affected by corrosion. To address this challenge, mobile manipulators have been developed to navigate these facilities, identify corroded areas, and apply protective coatings. However, given that this technology is still in its infancy and the consequences of improperly coating essential equipment can be significant, human oversight is necessary to review the robot's corrosion identification and repair plan. We present a practical and scalable Augmented Reality (AR)-based system designed to empower non-experts to visualize, modify, and approve robot-generated surface corrosion repair plans in real-time. Built upon an AR-based human-robot interaction framework, Augmented Robot Environment (AugRE), we developed a comprehensive AR application module called Situational Task Accept and Repair (STAR). STAR allows users to examine identified corrosion images, point cloud data, and robot navigation objectives overlaid on the physical environment within these industrial environments. Users are able to additionally make adjustments to the robot repair plan in real-time using interactive holographic volumes, excluding critical nearby equipment that might be at risk of coating overspray. We demonstrate the entire system using a Microsoft HoloLens 2 and a dual-arm mobile manipulator. Our future research will focus on evaluating user experience, system robustness, and real-world validation.Comment: Winning Paper (2nd Prize) at The Second International Horizons of an Extended Robotics Reality (XR-ROB) Workshop - IEEE IROS 2023 | Workshop Website: https://sites.google.com/view/xr-robotics-iros2023/home?authuser=

Multimodal Grounding for Embodied AI via Augmented Reality Headsets for Natural Language Driven Task Planning

Recent advances in generative modeling have spurred a resurgence in the field of Embodied Artificial Intelligence (EAI). EAI systems typically deploy large language models to physical systems capable of interacting with their environment. In our exploration of EAI for industrial domains, we successfully demonstrate the feasibility of co-located, human-robot teaming. Specifically, we construct an experiment where an Augmented Reality (AR) headset mediates information exchange between an EAI agent and human operator for a variety of inspection tasks. To our knowledge the use of an AR headset for multimodal grounding and the application of EAI to industrial tasks are novel contributions within Embodied AI research. In addition, we highlight potential pitfalls in EAI's construction by providing quantitative and qualitative analysis on prompt robustness.Comment: 18 pages, 15 figure

TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface

Teleoperated mobile robots, equipped with object manipulation capabilities, provide safe means for executing dangerous tasks in hazardous environments without putting humans at risk. However, mainly due to a communication delay, complex operator interfaces and insufficient Situational Awareness (SA), the task productivity of telerobots remains inferior to human workers. This paper addresses the shortcomings of telerobots by proposing a combined approach of (i) a scalable and intuitive operator interface with gestural and verbal input, (ii) improved Situational Awareness (SA) through sensor fusion according to documented best practices, (iii) integrated virtual fixtures for task simplification and minimizing the operator’s cognitive burden and (iv) integrated semiautonomous behaviors that further reduce cognitive burden and negate the impact of communication delays, execution latency and/or failures. The proposed teleoperation system, TeMoto, is implemented using ROS (Robot Operating System) to ensure hardware agnosticism, extensibility and community access. The operator’s command interface consists of a Leap Motion Controller for hand tracking, Griffin PowerMate USB as turn knob for scaling and a microphone for speech input. TeMoto is evaluated on multiple robots including two mobile manipulator platforms. In addition to standard, task-specific evaluation techniques (completion time, user studies, number of steps, etc.)—which are platform and task dependent and thus difficult to scale—this paper presents additional metrics for evaluating the user interface including task-independent criteria for measuring generalized (i) task completion efficiency and (ii) operator context switching

Validation of Near-Real-Time NOAA-20 CrIS Outgoing Longwave Radiation with Multi-Satellite Datasets on Broad Timescales

The Outgoing Longwave Radiation (OLR) package was first developed as a stand-alone application, and then integrated into the National Oceanic and Atmospheric Administration (NOAA) Unique Combined Atmospheric Processing System (NUCAPS) hyperspectral sounding retrieval system. An objective of this package is to provide near-real-time OLR products derived from the Cross Track Infrared Sounder (CrIS) onboard the Joint Polar Satellite System (JPSS) satellites. It was initially developed and validated with CrIS onboard the Suomi National Polar-orbiting Partnership (SNPP) satellite, and has been expanded to JPSS-1 (renamed NOAA-20 after launch) datasets that are currently available to the public. In this paper, we provide the results of detailed validation tests with NOAA-20 CrIS for large and wide representative conditions at a global scale. In our validation tests, the observations from Clouds and Earth’s Radiant Energy System (CERES) on Aqua were treated as the absolute reference or “truth”, and those from SNPP CrIS OLR were used as the transfer standard. The tests were performed on a 1°×1° global spatial grid over daily, monthly, and yearly timescales. We find that the CrIS OLR products from NOAA-20 agree exceptionally well with those from Aqua CERES and SNPP CrIS OLR products in all conditions: the daily bias is within ±0.6 Wm−2, and the standard deviation (STD) ranges from 4.88 to 9.1 Wm−2. The bias and the STD of OLR monthly mean are better, within 0.3 and 2.0 Wm−2, respectively. These findings demonstrate the consistency between NOAA-20 and SNPP CrIS OLR up to annual scales, and the robustness of NUCAPS CrIS OLR products

DE-STAR: Phased-Array Laser Technology for Planetary Defense and Other Scientific Purposes

Current strategies for diverting threatening asteroids require dedicated operations for every individual object. We propose a stand-off, Earth-orbiting system capable of vaporizing the surface of asteroids as a futuristic but feasible approach to impact risk mitigation. We call the system DE-STAR (Directed Energy System for Targeting of Asteroids and exploRation). DE-STAR is a modular phased array of laser amplifiers, powered by solar photovoltaic panels. Lowcost development of test systems is possible with existing technology. Larger arrays could be tested in sub-orbital demonstrations, leading eventually to an orbiting system. Design requirements are established by seeking to vaporize the surface of an asteroid, with ejected material creating a reaction force to alter the asteroid’s orbit. A proposed system goal would be to raise the surface spot temperature t